Suboptimality of Penalized Empirical Risk Minimization in Classification
نویسنده
چکیده
Let F be a set of M classification procedures with values in [−1, 1]. Given a loss function, we want to construct a procedure which mimics at the best possible rate the best procedure in F . This fastest rate is called optimal rate of aggregation. Considering a continuous scale of loss functions with various types of convexity, we prove that optimal rates of aggregation can be either ((logM)/n) or (logM)/n. We prove that, if all the M classifiers are binary, the (penalized) Empirical Risk Minimization procedures are suboptimal (even under the margin/low noise condition) when the loss function is somewhat more than convex, whereas, in that case, aggregation procedures with exponential weights achieve the optimal rate of aggregation.
منابع مشابه
Oracle Inequalities and Adaptive Rates
We have previously seen how sieve estimators give rise to rates of convergence to the Bayes risk by performing empirical risk minimization over Hk(n), where (Hk)k ≥ 1 is an increasing sequence of sets of classifiers, and k(n) → ∞. However, the rate of convergence depends on k(n). Usually this rate is chosen to minimize the worst-case rate over all distributions of interest. However, it would be...
متن کاملMultivariate Dyadic Regression Trees for Sparse Learning Problems
We propose a new nonparametric learning method based on multivariate dyadic regression trees (MDRTs). Unlike traditional dyadic decision trees (DDTs) or classification and regression trees (CARTs), MDRTs are constructed using penalized empirical risk minimization with a novel sparsity-inducing penalty. Theoretically, we show that MDRTs can simultaneously adapt to the unknown sparsity and smooth...
متن کاملSpring 2014 Statistics 210 b ( Theoretical Statistics ) - Lecture One Aditya
1. Some aspects of classical empirical process theory: uniform laws of large numbers, process convergence and uniform central limit theorems. 2. M-estimation. Asymptotic theory of consistency, rates of convergence and limiting distribution. 3. Non-asymptotic theory of penalized empirical risk minimization; nonasymptotic deviation inequalities for suprema of empirical processes, oracle inequalit...
متن کاملOracle Inequalities in Empirical Risk Minimization and Sparse Recovery Problems
A number of problems in nonparametric statistics and learning theory can be formulated as penalized empirical risk minimization over large function classes with penalties depending on the complexity of the functions (decision rules) involved in the problem. The goal of mathematical analysis of such procedures is to prove ”oracle inequalities” describing optimality properties of penalized empiri...
متن کاملFrom Dual to Primal Sub-optimality for Regularized Empirical Risk Minimization
Regularized empirical risk minimization problems are fundamental tasks in machine learning and data analysis. Many successful approaches for solving these problems are based on a dual formulation, which often admits more efficient algorithms. Often, though, the primal solution is needed. In the case of regularized empirical risk minimization, there is a convenient formula for reconstructing an ...
متن کامل